When processing the image captured by a television camera, knowledge of the motion of the edges and/or corners in the image is very useful as it facilitates the determination of the motion of the camera relative to objects in the image (up to a scale factor). Knowledge of the relative camera motion, defined as camera egomotion, is often useful for the purpose of image coding and/or image compression, and for computing the shape and three-dimensional position of objects in the image, up to the same scale factor mentioned earlier. The scale factor, which applies to both the size and depth of the object, is not determinable with a single camera. A knowledge of the location and shape of an object in an image is valuable, especially in robotics, for accomplishing autonomous navigation and manipulation.
The advent of digital technology, and especially digital image processing systems, has permitted the image of an object captured by a television camera to be sampled over both space (within the camera image plane) and time. Such spatiotemporal samples can be processed by the known techniques of intensity flow or edge flow, as described in greater detail below, to estimate the image velocity of visible object points. Calculation of the image velocity based on the intensity flow technique is predicated on the assumption that the spatiotemporal variation in the image intensity is continuously differentiable over time and space; and that for every visible point in space, its image intensity is time-invariant. Under these assumptions, the following expression can be derived for the component of the image velocity in the direction of the image intensity gradient. ##EQU1##
It should be noted that Eq. (1) is not applicable to constant-intensity image regions, nor does it yield any information about the image velocity component perpendicular to the direction of the image intensity gradient. In other words, the equation provides no information about the image velocity component in a direction tangent to the isobrightness contour through the image point. This difficulty is known in the art as the intensity-flow aperture problem. As may be appreciated, the intensity flow technique estimates image velocity by the flow of image isobrightness contours which, in general, is different from that of the imaged scene points. The principal disadvantage of the intensity flow technique is that the assumption on which it is based, namely, the scene radiance is time-invariant in the direction of the camera, holds only for Lambertian surfaces with time-invariant irradiance.
Rather than estimate the image velocity based on the flow of isobrightness contours, the image velocity of visible scene points can be estimated based on the flow of the intensity edges within the image. The advantage of estimating image velocity in this manner is that the pre-image of each of the intensity edges, that is to say, their location in real space, is in general more likely to be invariant (with respect to the imaged objects in the scene) under motion of the camera relative to the scene and with respect to changes in the scene illumination, as compared to the pre-image of each isobrightness contour. The intensity flow and edge flow techniques are, in a sense, complimentary because the intensity flow technique depends upon continuity in the image intensity, whereas the edge flow technique is applicable precisely where continuity is lacking, at the edges and corners of objects in the image.
Estimating the image velocity by detecting the flow of edges and/or corners in a stack or collection of images in so-called spatiotemporal image space is preferable to detecting edges and/or corners in single frames and then matching them between frames. This may be understood by appreciating that the spatiotemporal image of a point is a coherent structure of one higher dimension, namely, a line. Similarly, the spatiotemporal image of a curve is a surface, and that of a surface, a volume. Detecting a line in three-dimensional space is easier than detecting a point in two-dimensional space, and detecting a surface in three dimensions is easier than detecting a line in two dimensions. In this context, while the image velocity of a scene point that maps onto a locally discernible spatiotemporal image curve can easily be determined by projecting the tangent to the curve onto the image plane, the image velocity of a scene point whose spatiotemporal image can only be restricted to some discernible surface, rather than to a curve, has a one-degree-of freedom local indeterminacy. This inherent limitation, known in the art as the edge-flow aperture problem, is analogous to the intensity-flow aperture problem discussed previously.
Thus, there is a need for a technique to determine and track the edges and/or corners in an image to determine their image velocities. This need is particularly great for tracking corners, which are not susceptible to the edge-flow aperture problem discussed earlier.